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Abstract 

Available assessments of our graduate research strategies course were (i) traditional pre- and 
post-test evaluation instruments, and (ii) student evaluations. We felt a need for a third kind of 
assessment to measure any changes in attitude to research, though we did not expect detectable 
changes in a 15 week course. Students were asked to rank their perceived barriers to research in a 
research barrier questionnaire, issued to them at the beginning and end of the course. 

While attitudes usually change slowly, we did detect a statistically significant difference in the 
mean rankings of the two constructs ‘procrastination’ and ‘framing hypotheses’ at the 5% level. 
Surprisingly these constructs were more firmly embedded by the end of the course. We were 
comforted by Festinger’s (1957) Cognitive Dissonance Theory which has been modified to say 
that attitudes may change quickly when people are confronted by the reality that they are 
‘wrong’. Suggestions were made for improvement of the barrier questionnaire and the research 
strategies course. 



Introduction 



Assessments of the First Kind 
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When it came to evaluating students taking a graduate course on research strategies in education, 
we had only two kinds of assessment. The first kind was a traditional period of teaching, 
symbolized by X, followed by a post-test observation, symbolize by O. 

The evaluation design was therefore represented by 

X O 

In teaching and indeed in research, this design is recognized as faulty, because it does not 
determine the entry behavior 01 and readiness of the students to receive the treatment X. 

Evaluation should be a two part process (i) determining the degree to which objectives are 
obtained, and (ii) comparing the performance against norms. If the objectives are strictly- 
behavioral, specifying entry behavior and expected terminal behavior, the X O design needs 
additional pre-testing of the students’ entry qualifications and experience. Then this improved 
design would be represented as 

01 X 02 

Many times we can be excused for using X O. We all have to teach ‘too soon’, before we have 
sufficient information about our students. As early as 1800, Dr. Caleb Parry said that it was 
much more important to know what sort of patient has a disease than what sort of disease she/he 
has. However, because of pressures of the national curriculum, sooner or later we have to get on 
with the job of delivering X. 



Assessments of the Second Kind 
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In addition to the cognitive evaluations, we were asked to acquire student evaluations. These 
immediately make incursions into the affective, psychomotor and conative domains. Examples 
of affective evaluations are those which accept or criticize the workload, draw attention to 
conflicts between work and family, and refer to the quality of teacher charisma and interaction 
with students. Psychomotor evaluations may refer to lack of, or difficulties with, transport to and 
from the class, particularly in bad weather, or disabilities such as carpal tunnel syndrome. 
Conative (or ‘action’) evaluations may refer to procrastination. Types A and B Personality 
problems, or resistance to relinquishing the comfort zone of the classroom to go out and obtain a 
research sample. Student evaluations provide beneficial opportunity for the students to give vent 
to their frustrations, but in the worst scenario the worst students give the teacher the worst 
reports, as excuses for their own laziness. 



Can we do better than assessments of the first and second kind? 

For the first kind, we can add a cognitive pre-test to the X O model, elevating it to 

01 X 02 



This is rather harsh, because it will be exposing gaps in entry knowledge rather than motivating 
the students to start the course. We tried to mix the pretest with steerable student introductions. 
Given a free rein, naive students would talk freely about themselves, whereas more sophisticated 
students had learnt to be cautious in introducing themselves, and confined themselves to 
describing their cognitive achievements. By imposing structure and cues, they could be 
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encouraged to open up in the affective domain, for example saying whether they were shy or 
confident, introverted or extraverted, even sexual or sensual. In the psychomotor domain, unless 
they were physical education students they rarely revealed whether they were team or individual 
players. To enter the conative domain, they needed yet more encouragement to say whether they 
were procrastinators, or Type A or B personalities. 

While we learnt a lot about the students from these revelations, for this particular course we felt a 
need for a third kind of assessment, combining evaluation of ourselves and the students. The 
course had less content than skills because it aimed to provide the students with transferable 
research strategies to use later in teaching and counseling occupations or in doctoral research. A 
conative aim was to encourage students to exercise creative and divergent thinking to solve 
problems in action research - an ambitious aim if they had been the victims of much rote 
learning in earlier education. 



Literature Review 

The closest research of our concerns was an article titled “What Today’s College Students Say 
about Conducting Research in the Digital Age”, by Head and Eisenberg (2009). They conducted 
their research during the fall semester of 2008. They instigated eleven discussion groups on 
seven college and university campuses across the United States. They talked with 86 full-time 
students studying the humanities and social sciences in private colleges and universities, public 
colleges and universities, and community colleges. In their discussion groups they heard first- 
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hand accounts about what conducting research means, what the stages of research entails, and 
what techniques, strategies, and solutions students apply throughout the process. They found 
that no matter where students are enrolled, no matter what information resources they may have 
at their disposal, and no matter how much time they have, the abundance of information 
technology and the proliferation of digital information resources make conducting research 
uniquely paradoxical: “Research seems to be far more difficult to conduct in the digital age than 
it did in previous times”. 

As some of the students in our class agreed in cohesion with the students in this study, research 
surprisingly seems to be far more difficult to conduct in the digital age than it did in previous 
times (Head & Eisenberg, 2009, p. 2). 

In this study Head and Eisenberg also addressed factors of course-related research versus 
everyday life research. Course-related research coincides with your projects and classes and a 
topic you are researching. However, in everyday life research you are for example searching 
home remedies, the weather, where to travel - things that are usually more pleasing for people to 
research. Head and Eisenberg (2009) stated that they have identified four types of context that 
students reported they tried to obtain during the research process. These contexts are (1) the Big 
Picture, (2) Eanguage, (3) Situational, and (4) Information-Gathering. They said that the Big 
Picture context involved selecting and defining a topic, understanding multiple sides of an 
argument, and figuring out how the topic might best fit into the course curriculum. In was also 
stated that in many of the settings the students had different library facilities to go to; however, 
the students had said that a lot of times they weren’t sure what to cite. Head and Eisenberg 
(2009) recited the Eanguage context involved obtaining the meaning and use of language as an 
essential context in the research process for either course-related or everyday life research. The 
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Situational context involves setting the parameters of a topic, especially how efforts may fit into 
expectations and a set of surrounding circumstances. Then the last type of context they reported 
was the Information- Gathering context that involves finding, accessing, and securing relevant 
research resources that “satisfies” individual research needs. The students reported being 
overwhelmed and many were procrastinators by nature. One of the strategies talked about was 
using the reference librarians because they could help pinpoint what you needed to find in a 
book, finding a particular book or articles online. Some students even referred to reference 
librarians, as “sense-makers”, because they even helped to make the project/research and 
research results make better sense to them. Students even reported as using Wikipedia as a 
starting off point in their research because it could give you other ideas that were related to the 
topic, or to get your wheels turning on the subject, where before they couldn’t get a head start. 

Overall, Head and Eisenberg (2009) summed up their findings - 

“The students reporting to be challenged, confused, and frustrated by the research 
process, despite the convenience, relative ease, or obliquity of the internet. In their 
discussion sessions, frustrations included the effects of information overload and being 
inundated with resources, but more. Participants also reported having particular 
difficulty traversing a vast and ever-changing information landscape. Specifically, 
participants greatest challenges were related to finding the materials they desired, knew 
existed, and needed on a “just in time basis” 

The overall nature of the research done by Head and Eisenberg and ourselves shows that 
research today, in the digital age, is very confusing, frustrating, and difficult with the perceived 
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and unperceived barriers entailed, and that some professors don’t forecast the troubles likely to 
be encountered by the students doing the research. 

A national survey conducted by Perl and Kahn, (1981) found that demands of jobs, problems of 
learned models of research, or a shift of interests may affect subsequent research output. These 
two citations triggered the idea that we ought to determine the perceived and real barriers of the 
students to effective research. Dare we look for changes in attitude over the relatively short time 
windows of our 5- and 15- week research courses, knowing that attitudes change so slowly 
unless we are subjected to shock, or brutal life changes such as bereavement? 

There is some hope that attitudes may change if people clearly know that their present attitude is 
‘wrong’ for achievement of a highly desired goal. As early as 1957 Festinger developed his 
Theory of Cognitive Dissonance (Festinger, 1957). The theory is validated by withstanding the 
test of time. As a Construct, Cognitive Dissonance is best understood by an example - suppose 
you decide to buy a Chrysler instead of a Pontiac. You then avoid Pontiacs until the Chrysler is 
yours. Dissonance is rooted in unpleasant feelings when your belief contradicts your decision. 
Festinger’ s research progressed to discover that dissonance would in fact change attitudes over 
time, helping people to justify their decisions or actions even when they knew they were wrong. 

Secret, Ford and Rompf (2003) revealed complex student attitudes towards learning research 
strategies, using a large sample over a long period of four years. There was a mixture of positive 
and negative attitudes and an initial fear of statistical analysis. 

Fortunately the research experience of our own team linked to the clues in the previous citations 
to provide an assessment of the third kind - 
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Cleeton G. (1991) published a Ph.D. thesis ‘ Towards a new theory of learning barriers’, in 
which he classified perceived and real learning barriers, though this was focused on an 
Electronics course which targeted content rather than skills. He followed this up with an article 
in Perceptual and Motor Skills (Cleeton G., 1996) in which he generalized the theme by 
developing a terminology for learning barriers as perceived, anticipated, real, illusory and 
discovered. Perceived barriers are those which are felt almost by intuition before a course starts. 
They are often scars from previous experience. They firm up to become anticipated barriers. 
Hopefully they do not gel to the extent where Piaget in discussing anticipatory images said that 
anything we anticipate is more likely to occur. 

As a course proceeds students may find their perceptions and anticipations realized as real 
barriers, or hopefully dispelled as illusory barriers. Persistence of the real barrier consolidates it 
as a confirmed barrier. The worst barrier you can encounter is the discovered barrier, which was 
neither perceived nor anticipated. Recalling our earlier critique of Head and Eisenbergs’ report 
(2009) we noted reference to unperceived hasners, analogous to Cleeton G.’s discovered 
barriers. 

Another of our research team was a graduate (Eberle, M. A., 2009), who enrolled in our course 
in research strategies, and expressed a desire to submit her research project on its evaluation. To 
her we are also indebted for her acquisition and critique of Head and Eisenberg’s article. 

Our third member of the team (Cleeton, E., 2003) clarified problem solving to surmount 
blockages in the action research process. She had published doctoral and post-doctoral research 
on external representations by enhancing their actions in the conative domain. This was done by 
recording the ‘rough work’ done in solving spatial problems. She was able to illuminate the team 
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experience by steering our content-based research into strategy-based research. In this 
connection, Johnson (2009) has published a valuable book devoted entirely to action research by 
teachers in schools. 

Collecting these critiques and team experience we attempted to fill a gap in evaluation by 
making an assessment of the third kind, in which we would determine student attitudes at the 
beginning and end of the course and try to detect any possible shifts, though we expected these to 
be small, if they existed at all. 

Research Question and Hypotheses 



Do graduate research students change their perceptions of research barriers from the beginning to 
the end of the course? 

Alternative hypothesis - Research students will change their perceived learning barriers to 
research during a 15 week course on research strategies 

Null hypothesis - There will be no significant difference between the perceived learning barriers 
of research students to research from the beginning to end of a 15 week course in research 
strategies 



Research Design 

First it was necessary to collect perceptions of barriers to research. Over a period of eight years, 
anecdotal impressions of these barriers were collected. Seventeen of the most frequent 
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perceptions were listed. The students were asked to rank these at the beginning and end of the 
course. Open-ended questions were added to the ranking list to ask for any other perceived 
barriers and perceived strengths for research. The final barrier questionnaire follows - 
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Research barrier questionnaire. 



Rank these perceived barriers to research - ‘1’ for greatest perceived barrier 





‘Writers’ block’ 




Procrastination - delaying work 




Re-writing 




Editing 




Sentence construction 




Choosing a project 




Searching the Literature 




Framing your objectives 




Prioritizing objectives 




Framing hypotheses 




Keeping the whole project in mind while focusing on part of it 




Getting a sample 




Matching learning style to the project 




Time management 




Constructing measuring instruments - questionnaires, tests 




Statistics 




Graphics 





What other barriers do you think you will have? 




Sample 

The barrier questionnaire was completed anonymously at the beginning and end of a 15 week 
course on research strategies by 19 graduate education students. 



Analysis of results 



Descriptive statistics 
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The plan was to examine the mean barrier rankings of the perception constructs and give a 
general description of most and least perceived barriers. 

A correlation matrix was to be examined to look for distinction or redundancy of the barrier 
constructs. This means for example that if there were a high, significant positive correlation 
between constructs X and Y (for example ‘rewriting’ and ‘editing’), that these constructs would 
be so highly related that one of the constructs was redundant and could be pruned for a future 
questionnaire. Such processes had already refined the questionnaire to its present state. 



Inferential statistics 

The plan was to look for significant differences between the mean ranking of each construct at 
the beginning and end of the course. Preliminary measurements of skewness, kurtosis and 
variance of each construct pair (that is, for the same construct at the beginning and end of the 
course) would be made to check for approximate normality of the distribution pairs and 
approximate equality of variance. If these checks were satisfied, parametric t-tests for paired 
samples could be applied to look for significant differences in the means of each variable pair. If 
not, which would be more likely since the data types were ordinal rankings not continuously 
interval, then the non-parametric paired-sample Wilcoxon Signed Rank Test would be applied. 



Results 
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During the course 1 1 mean perceived barrier rankings were raised and 6 were lowered. A 
superficial conclusion would be that the course was effective in lowering barriers to research, but 
that depends on whether individual post-course to pre-course means were statistically significant. 
Since the data type was ordinal and deviated from normal, as frequently expected in a small 
sample, a non-parametric difference test was needed to look for significant differences between 
post- and pre-course mean rankings for each of the 17 perceived barrier constructs. The choice 
was between the paired sample sign test and the paired sample Wilcoxon signed rank test. While 
the paired sample sign test only uses the signs of the paired-difference ranks, the paired sample 
Wilcoxon signed rank test uses magnitude and sign of the paired difference ranks, so this was 
used. 

There was a significant difference between the mean ranking of the construct ‘Procrastination’ 
from beginning to end of the course at the 5% level, but the perception of its difficulty became 
worse. 

There was a significant difference between the mean ranking of the construct ‘’Framing 
hypotheses’ from beginning to end of the course at the 5% level, but the perception of its 
difficulty again became worse. 

These findings indicate that students finishing the course would perceive ‘procrastination’ as not 
only a serious barrier to future research, but moreover perceive it as ‘worse’ than at the 
beginning of the course! The same applies to ‘framing hypotheses’. 



Discussion 
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While it is always encouraging to find any statistically significant research results, they do not 
always go in the expected direction, but further examination of these often provide important 
insights when given careful consideration. At first it looks as though the course, or the action 
research conducted by the students, embedded fear of ‘procrastination’ and ‘framing 
hypotheses’. On the other hand, this was the first time that the students had left the comfort zone 
of the classroom to go out and do action research in the schools. In our first lesson with the 
students, we had warned them that procrastination was the enemy of research, but this lesson 
may only be learned by experience. So it may be a good thing that the students came to fear 
‘procrastination’ if they wish to do improved research in the future. After all, this course was not 
loaded with content, but was an introduction to research strategies. 

Similar considerations apply to the embedded perception of ‘framing hypotheses’. 



Raising the quality of our research experiment 

We can add more constructs to the research barrier rankings. These can be found in the answers 
to its open-ended questions. On the other hand we can prune redundant constructs from it. We 
made a correlation matrix of all the variable pairs and found for example that ‘rewrite’ and ‘edit’ 
were highly positively correlated (>0.65). Therefore we could eliminate either one of these 
constructs. 



Raising the quality of our research strategies course 
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We can capitalize on the knowledge that our course does not necessarily alleviate fear of 
‘procrastination’ and ‘framing hypotheses’. If there were time, maybe a course on critical path 
analysis would be beneficial, complete with float times at its nodes and risk analysis for non- 
completion. To improve the framing of hypotheses we may need to give the students more work 
on framing strictly-behavioral objectives. 

Festinger (1957) said that although attitudes in general change slowly, one way to accelerate 
them is to make it clear to the student that their attitude is ‘wrong’ . In this course some of the 
students certainly embedded ‘procrastination’ as a warning for next time. However it has been 
said that the purpose of doing a project is to do a better one next time. 
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